Advanced Lane Finding Project¶

The goals / steps of this project are the following:

Compute the camera calibration matrix and distortion coefficients given a set of chessboard images.
Apply a distortion correction to raw images.
Use color transforms, gradients, etc., to create a thresholded binary image.
Apply a perspective transform to rectify binary image ("birds-eye view").
Detect lane pixels and fit to find the lane boundary.
Determine the curvature of the lane and vehicle position with respect to center.
Warp the detected lane boundaries back onto the original image.
Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.

Computation of the camera calibration using chessboard images¶

First we need to calibrate the camera using chess board images taken by the same camera. It is imporant to calculate the lane curvature and car positon correctly.You can find the camaera calibration codes in the Camera.py. The Camera class expects the path of the calibration images (chess board images). Camera class undistort methods is used for calibrating the new images. First it calls calibrate method if the calibration matrix has't calculated yet it also checks if there is already saved data on the disk for the calibration parameters (mtx, dst) according to the image size. For calibrating the camera we go through the all calibration images and find the object and image points by using OpenCV findChessboardCorners method. After that we use the OpenCV calibrateCamera function with the found object and image points to calculate camera calibration and distortion coefficients. I applied this distortion correction to the test image using the cv2.undistort() function. You can see an example of camera calibration with one of the calibration image.

from Camera import Camera
import matplotlib.pyplot as plt
import cv2
img = cv2.imread('camera_cal/calibration1.jpg')
calibration_path = 'camera_cal'
camera  = Camera(calibration_path)
# Test undistortion on an raw calibration image

dst = camera.undistort(img)
cv2.imwrite('output_images/test_undist.jpg',dst)

# Visualize undistortion
f, (ax1, ax2) = plt.subplots(1, 2, figsize=(20,10))
ax1.imshow(img)
ax1.set_title('Original Image', fontsize=30)
ax2.imshow(dst)
ax2.set_title('Undistorted Image', fontsize=30)
plt.show()

Undistortion Example¶

In the cell bellow there is an example of camera calibration with one of the test image taken from the car camera.

#Test undistortion on an test image
img = plt.imread('test_images/test5.jpg')
figsize=(10,5)
dst = camera.undistort(img)
cv2.imwrite('output_images/test2.jpg',dst)

#dst = cv2.cvtColor(dst, cv2.COLOR_BGR2RGB)
# Visualize undistortion
f, (ax1, ax2) = plt.subplots(1, 2, figsize=figsize)
ax1.imshow(img)
ax1.set_title('Original Image', fontsize=25)
ax2.imshow(dst)
ax2.set_title('Undistorted Image', fontsize=25)
plt.show()

Thresholded Binary Image¶

In the cell bellow there is an example of masking image to mask lane lines. In order to create the binary image to mask the lane information several image processing teqniues were tested and combined. In the ImageProcesing.py you can find the several methods to create maksed image for the lanes. We tested direction thresholding on x and y magnitude thresholding with different parameters. In the end we used a combination of sobel threshold on x direction and HLS color space threshold. First the image is converted to the grey scale before calculation of the gradient. Then we used cv2.Sobel() method on x direction only to calculate the gradient and masked the image by only accepting the values between 30 and 150. Also image is converted to the HLS color space and pixels selected if the S channel value is between 175 and 250. After that bitwise or operation is applied to the masked images. (sobelx and extracred S). In ImageProcesing.py combined_threshold method is used to create the thresholded binary image and you can find an example of it on the cell bellow.

def abs_sobel_thresh(img, orient='x', thresh=(0, 255)):
    if orient == 'y':
        sobel = cv2.Sobel(img, cv2.CV_64F, 0, 1)
    else:
        sobel = cv2.Sobel(img, cv2.CV_64F, 1, 0)
    # 3) Take the absolute value of the derivative or gradient
    abs_sobel = np.absolute(sobel)
    # 4) Scale to 8-bit (0 - 255) then convert to type = np.uint8
    scaled_sobel = np.uint8(255 * abs_sobel / np.max(abs_sobel))
    # 5) Create a mask of 1's where the scaled gradient magnitude
    # is > thresh_min and < thresh_max
    sbinary = np.zeros_like(scaled_sobel)
    sbinary[(scaled_sobel >= thresh[0]) & (scaled_sobel <= thresh[1])] = 1
    # plt.imshow(sbinary, cmap='gray')
    # 6) Return this mask as your binary_output image
    return sbinary

def SThreshold(img, thresh=(0, 255)):
    # apply thresholding
    s_thresh = cv2.inRange(convert2S(img).astype('uint8'), thresh[0], thresh[1])
    s_binary = np.zeros_like(s_thresh)
    s_binary[(s_thresh == 255)] = 1
    return s_binary

def combined_threshold(image):
    ksize = 3  # Choose a larger odd number to smooth gradient measurements
    gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
    gradx = abs_sobel_thresh(gray, orient='x', thresh=(30, 150))
    sbinary = SThreshold(image, thresh=(175, 250))
    combined = np.zeros_like(gradx)
    combined[(gradx == 1) | (sbinary == 1)] = 1
    return combined

from ImageProcesing import combined_threshold

img = plt.imread('test_images/test5.jpg')

undistorted_image = camera.undistort(img)
combined_image = combined_threshold(undistorted_image)
# Plot the result
f, (ax1, ax2) = plt.subplots(1, 2, figsize=figsize)
f.tight_layout()
ax1.imshow(img,cmap='gray')
ax1.set_title('Original Image', fontsize=25)
ax2.imshow(combined_image, cmap='gray')
ax2.set_title('Masked Image', fontsize=25)
plt.subplots_adjust(left=0., right=1, top=0.9, bottom=0.)
plt.show()

Perspective Tranform¶

In the masked image above there are a lots of pixels seen other than lanes.In order to clear them we need to apply perspective transform to get interested area on the image.Thus transforming the image to another view (bird's eye view) is important to find lane lines and caculate the properties easily. For that pupose we implemented PerspectiveTransform class on PerspectiveTransform.py. It uses pre defined source and destination points (see bellow) and calculates the transformation and invers transformation matrixes by using cv2.getPerspectiveTransform() method. With the calculated matrixes we can transform a given image to specified perspective by using cv2.warpPerspective() method. In the cell bellow there is an example of perpective transform to birdseye view with one of the test image.

 - hard coded source and destination points
     src = np.float32([(130, 700), (540, 470), (740, 470), (1150, 700)])
     dst = np.float32([(330, 720), (330, 0), (950, 0), (950, 720)])

from PerspectiveTransform import PerspectiveTransform
perspectiveTransformer = PerspectiveTransform()
img = plt.imread('test_images/test5.jpg')

undistorted_image = camera.undistort(img)
transformed_image = perspectiveTransformer.get_perpective_transform(undistorted_image)
# Plot the result
f, (ax1, ax2) = plt.subplots(1, 2, figsize=figsize)
f.tight_layout()
ax1.imshow(img,cmap='gray')
ax1.set_title('Original Image', fontsize=25)
ax2.imshow(transformed_image, cmap='gray')
ax2.set_title('Transformed Image (Birdseye )', fontsize=25)
plt.subplots_adjust(left=0., right=1, top=0.9, bottom=0.)
plt.show()

In the cell bellow you can see and example of output of implemented teqnies so far (camera calibration, perspective trasnfor and lane masking). As seen on the left image still is not perfecly masked there are some noise and non lanes pixels we will deal with that with the Lane detector module to detect only the lanes and put it on the image.

img = plt.imread('test_images/test5.jpg')

undistorted_image = camera.undistort(img)
perspectiveTransformer = PerspectiveTransform()
transformed_image = perspectiveTransformer.get_perpective_transform(undistorted_image)
combined_image = combined_threshold(transformed_image)
# Plot the result
f, (ax1, ax2) = plt.subplots(1, 2, figsize=figsize)
f.tight_layout()
ax1.imshow(img,cmap='gray')
ax1.set_title('Original Image', fontsize=25)
ax2.imshow(combined_image, cmap='gray')
ax2.set_title('Transformed Masked Image', fontsize=25)
plt.subplots_adjust(left=0., right=1, top=0.9, bottom=0.)
plt.show()

Lane Detection¶

Lane.py¶

We implemented two class for the lane detection. The Lane class (on Lane.py) is implemented to represent lanes. It has several attributes for lane parameters. It calculates the lane polynome (current_fit) by given x and y points. Also it keeps track of the previously found lane parameters to generate better results (it calculates the best_fit by using the old lane parametes up to 10 previuos frames.) (smoorthing etc.). It also has methods to check and compare the lanes.

LaneDetector.py¶

LaneDetector class has the processing and searrch methods. In the init method of LaneDetector class first we calculate the Camera calibration and transformation matrixes. First the process_image method is callled there the image is undistorted and pass to detect_lanes method. On the detect_lanes method first we apply combined_threshold then transformed it to get binary warped image to strart searching for the lanes. Then we cut the image half on the x axis to search left and right lane seperately. We implemented two methods to search for the lanes the first one is histogram_search and the second one is search_along_previous_lane.

histogram_search ( LaneDetector.py line 193-260)¶

First we cut the image (we pass left and right halves then cut the half by two on the y axis so acctually we create the histogram of the quarter of the original image.) on y axsis and create the histogram. Then we can use the point where the histogram has peak for the starting point of the sliding window search from top the bottom of the image. You can see the found windows and points on for the left (red points) and the right lanes.

Combined Image — Sliding windows of the histogram_search for the left lane.

histogram_search method can detect the lanes fairly well however it takes around 5 seconds to make the search. 5 secodns is not acceptable for the real time purposes. Thus we implemented second search method to speed up the process.

search_along_previous_lane ( LaneDetector.py line 263-292)¶

After detecting the lanes in the previous frame we don't need to do search blindly as the lanes location don't change a lot from one frame to another, for that purpose we use search_along_previous_lane method. It uses the lanes found in the previous frame and make do the search near them (with a margin equals 100).

detect_lanes ( LaneDetector.py line 95-137)¶

This method checks the previosly found lanes to decide which search method will be used. If the system can found the lanes and validated them according to their parameters (see the validate_lane method) the search_along_previous_lane will be used to locate the lanes otherwise system will start the histogram_search to do the complete search again. After the search draw_lanes is called to paint the lanes area and draw_lanes_info to print the lanes info (radius, distance from the center and x locations).

validate_lane ( LaneDetector.py line 139-177)¶

It check whether the found lanes is make sense by first comparing them to each other. For that first we calculated the new lane polynoms and we check the first and second coefficients of the polynomes to decide if they are parallel ( is_parallel in Lane.py line 68-75).Also we calculate the distance between the lanes by using calculate_distance (Lane.py line 77-83) . If the lanes are parellel and the distance is in between thresholds than we said that the both lanes are ok and return. Otherwise we compare teh left and the right lane by the previous coefficinets and validate them seperately. (That means we can validate only one lane and not the other one on a frame).

draw_lanes ( LaneDetector.py line 55-92)¶

It uses the best fit polynomial of the found lanes and creates points along the y axis. Then it uses cv2.fillPoly method to fill the lane area. We also draw lines (red lines for the left and blue for the right) on the lanes by using cv2.polylines method. After drawing on the warped image we use get_reverse_transform method of the PerspectiveTransform to get lanes overlay on the original shape. finaly we add the lane overlay to the original image by using cv2.addWeighted method.

draw_lanes_info ( LaneDetector.py line 31-53)¶

This method prints the lane information on to frame. You can see the disnance from the center on the center above and left and right lanes informations (radius and the position) on the left and right bottom respectively.It uses calculate_radius method of the Lane class to calculate the lanes radius. We use the lanes base coordinates the calculate the distance from the center.First we calculate the lane center in pixel values by simply adding the left and rigth x coordinate and divide it by two. Than we compare it the center of the image then we convert the distance from thecenter from pixel values to real world value.

    center = image.shape[1] / 2
    lane_center = self.right_line.line_base_pos - self.left_line.line_base_pos
    center_x = (lane_center / 2.0) + self.left_line.line_base_pos
    cm_per_pixel = 370.0 / lane_center  # US regulation lane width = 3.7m
    dist_from_center = (center_x - center) * cm_per_pixel

calculate_radius ( Lane.py line 85-108)¶

We use the conversion bellow to calculate the real radius of the lanes. On this project we make calculations all in pixel coordinates and it is different than the real life. thus we define a converion parameters to convert the pixel values in real life information. After that we compute the second order polynomial of the lanes in meters. Lets say

    - f(y) = Ax^2+Bx+C

is the second order polynome of the found lane. We can calculate the radius by simply using first and the second order derivatives of the curve.

Raius Formula — The radius of curvature of the function x=f(y)

Lane Detection on Test Images¶

You can see the output of the pipeline on the cell bellow.

import glob
import time
from LaneDetector import LaneDetector
from Lane import Lane
import matplotlib.image as mpimg
import matplotlib.pyplot as plt

images = glob.glob('test_images/*.jpg')

for image_path in images:
    image = mpimg.imread(image_path)
    # create the lane detector object to search for the lanes 
    laneDetector = LaneDetector()
    t = time.time()
    output = laneDetector.process_image(image)
    t2 = time.time()
    print(round(t2 - t, 2), 'procesing time ')
    
    fig = plt.figure()
    plt.subplot(121)
    plt.imshow(image)
    plt.title('Original Image .')
    plt.subplot(122)
    plt.imshow(output)
    plt.title('Output.')
    fig.tight_layout()
    plt.show()

/home/burak/git-ws/Advance_Lane_Finding/CarND-Advanced-Lane-Lines/LaneDetector.py:197: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  histogram = np.sum(half[half.shape[0] / 2:, :], axis=0)

5.69 procesing time

5.64 procesing time

5.69 procesing time

5.92 procesing time

5.67 procesing time

5.82 procesing time

5.59 procesing time

5.89 procesing time

Pipeline (video)¶

You can run the pipeline on the "project_video" by simply running the cell bellow. It is already on the repository you can wath it on the two cell bellow or on this link

import os
from moviepy.editor import VideoFileClip

laneDetector = LaneDetector()
white_output = 'project_video_out.mp4'  # New video
#os.remove(white_output)
clip1 = VideoFileClip('project_video.mp4')  # .subclip(21.00,25.00) # project video
# clip = VideoFileClip("myHolidays.mp4", audio=True).subclip(50,60)
white_clip = clip1.fl_image(laneDetector.process_image)  # NOTE: this function expects color images!!
white_clip.write_videofile(white_output, audio=False)

[MoviePy] >>>> Building video project_video_out.mp4
[MoviePy] Writing video project_video_out.mp4

100%|█████████▉| 1260/1261 [02:49<00:00,  7.70it/s]

[MoviePy] Done.
[MoviePy] >>>> Video ready: project_video_out.mp4

from IPython.display import HTML
HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(white_output))

Discussion¶

The fisrt problem I have is to create the thresholded binary image to maske the lane lines. We can have better binary image by using the other teqnies like direction and magnitute thresholding. Morever we can also use Canny and Hough Lines to detect the lanes better. (I will try to use the same pipeline on the first project and chek the result. )

I also have problems during the video processing I use the histogram and sliding window search on the test images and it was working well then I saw that using the histogram search on every frame make the pipeline very slow then I implemented the second search as explanied in the course by searching only near the previously found lanes. However even with the histogram search sometimes the pipeline coudn't find the lanes correclty (see the second example on the Lane Detection on Test Images section). For that problem maybe we can use annother sliding window method by applying a convolution, which will maximize the number of "hot" pixels in each window in parallel to the histogram search. (Pool class can be used to make it paralel). Therefore we can use the best output from the both search methods and improve the system.

Also I tuned the lanes validation parameters very stritly to get rid of the false detections and jittering and that causes failures on the chalange videos. In order to achieve this I need to change the color trasnformations or maybe I need to use Hough Lines to have complete the chalenge videos.